Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Fix build with system's openmp #15369

Closed
wants to merge 2 commits into from
Closed

Fix build with system's openmp #15369

wants to merge 2 commits into from

Conversation

hubutui
Copy link

@hubutui hubutui commented Jun 26, 2019

Description

This pr fix build with openmp by:

  1. find openmp using FindOpenMP module from cmake
  2. if not found, use build openmp from source using 3rd party submodule.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
  • Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
  • Code is well-documented:
  • For user-facing API changes, API doc string has been updated.
  • For new C++ functions in header files, their functionalities and arguments are documented.
  • For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
  • Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
  • To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

This pr is a minor modification of CMakeLists.txt, simply find openmp firstly in the system, and then build from source if not found.

cmake 3.0.2-3.7 set OPENMP_FOUND
cmake 3.9-3.15.0.rc2 set OpenMP_FOUND
from cmake 3.9.5, OpenMP_<lang>_FOUND is preferred
@marcoabreu
Copy link
Contributor

Sounds less like a fix but rather a change in the logic if I'm not mistaken.

It would be good to have performance numbers before we move forward with this PR.

@larroy @lebeg @cjolivier01

@lebeg
Copy link
Contributor

lebeg commented Jun 26, 2019

I'm afraid that this change is rather a complicated one due to long discussions that have not ended yet. You can find more info on the dev mailing list: [Discussion] Remove bundled llvm OpenMP. Initially the proposed change was in this PR #12160: Remove conflicting llvm OpenMP from cmake builds .

We have already provided performance data on this, you can find and announcement on dev here and the document here.

That was not enough, for reasons that are unclear to me, to convince everybody on validity and necessity of this change.

Latest discussions happen in following dev threads: OMP and [VOTE] Remove conflicting OpenMP from CMake builds.

@hubutui If you feel motivated enough to proceed with this change you can join the debate.

@hubutui
Copy link
Author

hubutui commented Jun 26, 2019

@lebeg Hmm, actually, do user really care that much about performance affected by different OpenMP? As a packager's opinion, I want a good way to create this package. Maybe we could give user an option to choose the OpenMP, just like BLAS.

@lebeg
Copy link
Contributor

lebeg commented Jun 26, 2019

Yes, it might be a good idea, but the numbers as stated in the performance doc don't differ that much. Managing this dependency within MXNet isn't a good idea at all in my opinion.

@hubutui
Copy link
Author

hubutui commented Jun 26, 2019

It's been a long time since the discussion. I'll remove 3rdparty/openmp before building, so that cmake would use openmp from system instead.

@anirudhacharya
Copy link
Member

@mxnet-label-bot add [pr-awaiting-review]

@marcoabreu marcoabreu added the pr-awaiting-review PR is waiting for code review label Jun 28, 2019
@larroy
Copy link
Contributor

larroy commented Jul 2, 2019

@hubutui would it be possible to add a CMake option to choose the OpenMP implementation that one wishes for? There's three options, MKL, 3rdparty OpenMP and the one found in the platform. So I think the pr goes in the good direction but we would need a commandline option.

@cjolivier01
Copy link
Member

cjolivier01 commented Jul 2, 2019 via email

@hubutui
Copy link
Author

hubutui commented Jul 2, 2019

@cjolivier01 #9686 suggests transitioning fully to cmake, it would help a lot. Maybe dropping the Makefile someday?

@cjolivier01
Copy link
Member

cjolivier01 commented Jul 2, 2019

Using cmake is another subject altogether. I think you're missing the point.
How is this PR not basically what was already blocked?

@cjolivier01
Copy link
Member

cjolivier01 commented Jul 2, 2019

This is basically a duplicate of turning off llvm openmp.

@cjolivier01 cjolivier01 closed this Jul 2, 2019
@larroy
Copy link
Contributor

larroy commented Jul 2, 2019

But we are distributing pip wheels with libgomp, at least we should be able to replicate that for development purposes.

@szha
Copy link
Member

szha commented Jul 2, 2019

the reason is that there is mismatch between cmake and make builds. pip is currently using make for building libmxnet. for now if you want to replicate that you can use make

@larroy
Copy link
Contributor

larroy commented Jul 3, 2019

@szha understood. it would be beneficial to be able to replicate that with CMake by having a selector for the openmp lib until then.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
pr-awaiting-review PR is waiting for code review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants